Apparatus, method and program for image search

ABSTRACT

One or more representative images extracted from an image group comprising a plurality of images is/are displayed. A part or all of the representative image or images, such as a main subject region or a background region including a search target, is/are selected from the representative image or images, and used for setting search conditions. The image group is searched for an image or images agreeing with the search conditions having been set.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.14/970,381, filed on Dec. 15, 2015, which is a continuation of U.S.patent application Ser. No. 13/710,184, filed on Dec. 10, 2012 andissued as U.S. Pat. No. 9,245,195, which is a continuation of U.S.patent application Ser. No. 11/540,555, filed on Oct. 2, 2006 and issuedas U.S. Pat. No. 8,358,902, which claims priority from JP 2005-286779,filed Sep. 30, 2005, and JP 2006-255163, filed Sep. 21, 2006, all ofwhich are incorporated in their entireties herein by reference.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates to an image search apparatus and an imagesearch method for searching a plurality of images for an image by usingrepresentative images extracted from the plurality of images. Thepresent invention also relates to a program that causes a computer toexecute the image search method.

2. Description of the Related Art

Since digital cameras do not need film unlike film cameras, digitalcameras have reduced the cost of photography itself. As a result, moreimages tend to be photographed by digital cameras. Increase in storagecapacity of recording media to be inserted therein also contributes tothis trend. Consequently, large numbers of images are stored inrecording media for digital cameras and in recording media such as harddiscs of personal computers or CD-Rs in which image data read from therecording media for digital cameras are stored. Some of the images areunnecessary images although the others are necessary. In addition, theimages are left unorganized in many cases. The necessary images refer togood-looking images while the unnecessary images refer to imagesphotographed unsuccessfully or images photographed twice or more, forexample.

Classifying such a large amount of images is a troublesome task.Therefore, an apparatus has been proposed for supporting a search of astock of images to extract a necessary image, extraction and deletion ofan unnecessary image from the stock, and classifying and organizing theimages based on events, dates, and locations, for example.

U.S. Pat. No. 6,999,636 describes the steps of calculating acharacteristic quantity of a search key image illustrated by a user tolook similar to a desired image, searching for a similar image based onthe characteristic quantity, and displaying the similar image foundthrough the search, as a manner of supporting an image search.

Furthermore, Japanese Unexamined Patent Publication No. 2002-108913describes the steps of defining a plurality of image parts, preparingvariations for the respective image parts, constructing an image to beregistered with a database by selecting one of variations for each ofthe image parts, and setting an image search condition at the time of animage search by selection from the image parts.

Meanwhile, in the case where a user searches a large amount of storedimages for a desired image, the user sets a search condition based onincomplete memories of the image.

Considering this fact that users carry out searches based on fragmentarymemories, it is substantially difficult for users to illustrate entireimages to search for, as has been described in U.S. Pat. No. 6,999,636.In addition, since skills of illustration vary from person to person, aconsiderable number of users do not think illustrating is a goodinterface.

In the case where a search condition is set according to selection fromprepared parts as has been described in Japanese Unexamined PatentPublication No. 2002-108913, the parts may not fit well to rememberedparts in actual images.

SUMMARY OF THE INVENTION

The present invention has been conceived based on consideration of theabove circumstances. An object of the present invention is therefore toprovide an image search apparatus and the like that enable an imagesearch more accurately and easily based on fragmentary memories of animage remembered by a user.

An image search apparatus of the present invention comprises:

display means for displaying one or more representative images extractedfrom an image group comprising a plurality of images;

search condition setting means for setting search conditions by using apart or all of the representative image or images selected therefrom;and

search means for searching the image group for an image agreeing withthe search conditions having been set.

The search condition setting means may also set a position in the partof the representative image or images as one of the search conditions.

The image search apparatus of the present invention may further compriseregion extraction means for extracting a main subject region and/or abackground region from the representative image or images as a candidateor candidates for selection of the part or all of the representativeimage or images and for causing the extracted region or regions tobecome selectable.

The search condition setting means may have a function for carrying outimage processing on the part or all of the representative image orimages having been selected. In this case, the search condition settingmeans sets the search conditions by using the part or all of therepresentative image or images having been subjected to the imageprocessing.

Furthermore, the search condition setting means may add search priorityas one of the search conditions to the part or all of the representativeimage or images having been selected.

An image search method of the present invention comprises the steps of:

displaying one or more representative images extracted from an imagegroup comprising a plurality of images;

setting search conditions by using a part or all of the representativeimage or images selected therefrom; and

searching the image group for an image agreeing with the searchconditions having been set.

The representative image or images refer(s) to an image or images thatbest represent(s) the type of an event or a main subject. For example,the representative image or images may be an image having been judged tobe in the best quality, an image having been judged to have the bestquality in a group classified according to distribution of acharacteristic quantity, an image having the highest result of scenejudgment, an image of the scene having the highest frequency, an averageimage according to the scene judgment, an image closest to the averageimage, an image at the center of distribution of a characteristicquantity in a group classified according to the distribution, and animage closest to the center, for example. The representative image orimages can also be an image wherein a specific person is shown in thelargest size, an image wherein a specific person is shown in thesmallest size, and an image wherein the face of a person is shown in thesmallest size enabling easier identification of a location representedby a background, for example.

The part of the representative image or images refers to a region of amain subject or a background such as a person, a human face, clothes, ananimal, a plant, a still object, a vehicle, a building, and a mountain.The part of the image or images may have been extracted. Alternatively,a user may manually extract a region including a boundary of the part.Furthermore, the user may manually select a region so that the selectedregion can be extracted as the part. In addition, the user may manuallyselect an outline region including a main subject or a background sothat a region of the main subject or the background can be extractedfrom the selected region.

The image search method of the present invention may be provided as aprogram that causes a computer to execute the method.

According to the image search apparatus and the image search method ofthe present invention, the search conditions are set by using the partor all of the representative image or images selected from therepresentative image or images extracted from the image group. In thecase where an image is searched for based on fragmentary memoriesthereof, the fragmentary memories are of something that exists in theimage group to be searched and probably appears frequently in the imagegroup. Furthermore, the fragmentary memories are highly likely composedof a combination of real image parts in the image group. Therefore,general image parts may not fit well to actual image parts that the userremembers, and a search using such general parts is inefficient. On theother hand, in the present invention, the part or all of therepresentative image or images can be used as an image part for thesearch, and the search can be carried out more easily with higheraccuracy based on the fragmentary memories of the image remembered bythe user.

In the case where the position in the part of the image or images isused as one of the search conditions, a position of a main subject orthe like, that is, a memory on composition of the image, can be used asthe search condition. Therefore, the image search can be carried outbased on the more detailed conditions.

Furthermore, if the region extraction means also comprises the imagesearch apparatus that extracts the main subject region and/or thebackground region from the representative image or images as thecandidate or candidates for selecting the part or all of the image orimages and causes the extracted region or regions to become selectable,the user simply has to carry out selection from the region or regionshaving been extracted. Therefore, the search conditions can be set byuse of a user interface that is easier to operate. This is especiallyeffective in the case where the present invention is applied to aterminal or the like installed in a store for general consumers.

In the case where the search condition setting means has the functionfor carrying out image processing on the part or all of therepresentative image or images having been selected and sets the searchconditions by using the part or all of the representative image orimages having been subjected to the image processing, the searchconditions can reflect an intention of the search more clearly.

Alternatively, in the case where the search condition setting means addsthe search priority as one of the search conditions to the part or allof the representative image or images having been selected, a searchprioritizing only a specific person can be carried out. Therefore, adegree of clearness of memory can be reflected in the search.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an external oblique view of an image search apparatus of afirst embodiment of the present invention;

FIG. 2 is a block diagram showing the configuration of the image searchapparatus in the first embodiment;

FIG. 3 is a flow chart showing procedures carried out in the firstembodiment;

FIG. 4 shows an example of a search screen;

FIG. 5 shows an example of a search result screen;

FIG. 6 is a block diagram showing the configuration of an image searchapparatus of a second embodiment of the present invention;

FIG. 7 is a flow chart showing procedures carried out in the secondembodiment;

FIG. 8 shows examples of extracted regions;

FIG. 9 shows an example of displaying an extracted region in arepresentative image after adding a bounding outline and the typethereto;

FIG. 10 shows an example of a search screen; and

FIG. 11 shows an example of a search result screen;

FIG. 12 shows an example of a search screen;

FIG. 13 shows an example of a priority setting window; and

FIG. 14 shows an example of a search result screen.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Hereinafter, embodiments of the present invention are described withreference to the accompanying drawings. FIG. 1 is an external obliqueview of an image search apparatus of a first embodiment of the presentinvention while FIG. 2 is a block diagram showing the configuration ofthe image search apparatus. As shown in FIG. 1, an image searchapparatus 1 in this embodiment is installed in a store such as a DPEstore for enabling users to carry out image searches and to place printorders. The image search apparatus 1 comprises various kinds of cardslots 4, a display unit 6, and an input unit 7. The card slots 4 areused for inserting therein various types of memory cards 2 that storeimages used at the time of print order placement, for reading the imagesfrom the memory cards 2, and for recording images in the memory cards 2.The display unit 6 comprises a liquid crystal monitor or the like usedfor various kinds of display for image searches and print orders. Theinput unit 7 comprises a mouse for inputting various kinds ofinstructions to the image search apparatus 1. The image search apparatus1 is connected to a printer 8 installed in the DPE store and to acontroller 10 comprising a computer that can carry out image processingand print order management.

As shown in FIG. 2, the image search apparatus 1 comprises a CPU 12 anda system memory 14, in addition to the display unit 6 and the input unit7 described above. The CPU 12 carries out various kinds of control suchas recording control and display control regarding image datarepresenting images, and controls each of the components comprising theimage search apparatus 1. The system memory 14 comprises a ROM thatstores a basic program for operating the image search apparatus 1 andvarious kinds of constants, and a RAM used as workspace for the CPU 12.

The image search apparatus 1 also comprises a compression/decompressionunit 22, a hard disc 24, a memory control unit 26, a display controlunit 28 (display means), and a network interface 30, in addition to thecard slots 4. The compression/decompression unit 22 compresses imagedata according to a format such as JPEG, and decompresses compressedimage data. The hard disc 24 stores image data read from the memorycards 2 and various kinds of programs executed by the CPU 12 such as aprogram for representative image extraction and image search. The memorycontrol unit 26 controls the system memory 14, the card slots 4, and thehard disc 24. The display control unit 28 controls display on thedisplay unit 6. The network interface 30 is used for connecting theprinter 8 and the controller 10 to the image search apparatus 1.

The card slots 4 correspond to the various types of the memory cards 2.However, only one card slot is shown as the card slot 4 in FIG. 2.

The image search apparatus 1 further comprises a representative imageextraction unit 32 for extracting one or more representative images aswill be described later, a search condition setting unit 34 (searchcondition setting means) for setting image search conditions, and asearch unit 36 (search means) for carrying out an image search accordingto the conditions having been set.

Hereinafter, functions of the representative image extraction unit 32,the search condition setting unit 34, and the search unit 36 aredescribed together with procedures carried out in this embodiment.

FIG. 3 is a flow chart showing the procedures of image search carriedout in this embodiment. The memory card 2 brought by a user stores imagedata of a plurality of images comprising an image group. When the userinserts the memory card 2 in the corresponding one of the card slots 4,the image search apparatus 1 reads the image data of the image groupstored in the memory card 2 (Step ST1), and temporarily stores the imagedata in the RAM of the system memory 14 or in the hard disc 24.

According to processing by the CPU 12, the image search apparatus 1displays a menu screen on the display unit 6 for enabling the user tocarry out selection from various kinds of services such as image searchand image printing, and let the user select a desired one of theservices (Step ST2). In the case where the user has selected any one ofthe services other than the image search service, the image searchapparatus 1 carries out the selected service (Step ST3). In the casewhere the user has selected the image search service, the representativeimage extraction unit 32 automatically extracts one or morerepresentative images from the images having been read (Step ST4).

The representative image or images refer(s) to an image or images thatbest represent(s) the type of an event or a main subject. As a method ofextraction of the representative image or images, any known method canbe adopted. For example, similar scenes comprising main subjects such aspeople and backgrounds are firstly grouped together according to acharacteristic vector representing a color, a shape, a size, and thelike, as has been described in Japanese Unexamined Patent PublicationsNo. 2003-330941 and 2005-049968. Image quality of each of the images ineach of the image groups is judged based on information on edges anddensity therein, and the image having the best quality is selected asthe representative image for the corresponding group.

The display control unit 28 displays a search screen on the display unit6 (Step ST5) after extraction of the representative image or images.FIG. 4 shows an example of the search screen. As shown in FIG. 4, asearch screen 40 has a representative image area 42 where a catalog ofrepresentative images 41 is displayed, an input area 43 for display ofan image used for a search, a “Search” button 44 for carrying out thesearch, and a “Clear” button 45 for clearing input data. In FIG. 4, therepresentative images are collectively shown by the reference code 41,and two of the representative images are referred to as images 41A and41B.

The user selects the representative image including a search target fromthe representative images 41 displayed as the catalog, and selects apart or all of the selected representative image as a regionrepresenting the search target. The user pastes the selected region inthe input area 43 to generate the image used for the search (hereinafterreferred to as the search image), and the search condition setting unit34 sets the search image as the search conditions (Step ST6). In otherwords, a part or all of the selected representative image is used as animage part for the search (hereinafter referred to as the search imagepart). The search image is generated in the following manner, forexample.

In the case where the search target is a main subject such as a person,an animal, and a building, the user selects the region by a dragoperation of the input unit 7 to include the search target in therepresentative image. The user then selects “Select as Main Subject”from a menu displayed by right clicking, and pastes the selected regionin the input area 43 by a drag-and-drop operation. At this time, theuser can paste the region by specifying a position of the region in thesearch image. The user can also adjust a size of the selected region inthe search image.

After selection of “Select as Main Subject”, it is preferable for aregion of the main subject to be detected along an outline thereof inthe region selected by the drag operation, according to a known methodsuch as edge detection. Only the region of the detected main subject isthen pasted in the input area 43. Alternatively, the user may manuallyextract the main subject along the outline thereof, by use of the inputunit 7. Furthermore, the image search apparatus 1 may automaticallyextract the region by using an edge detection method or the like afterthe user clicks on the main subject. The user may select not only theentire person but also a part thereof, such as the face or clothes.

In the case where the search target is a background, the user selects aregion of the background as the search target by clicking on or draggingthe region in the representative image including the target, and selects“Select as Background” from the menu displayed in response to rightclick. The user selects a region in the input area 43 to be used for thebackground by a drag operation, and selects “Paste the SelectedBackground” by right click. In this manner, the selected background ispasted in the selected region in the input area 43. For the background,the user can also specify a position and a size thereof in the searchimage. In the case where the entire representative image is the searchtarget, the entire representative image is selected by a click or a dragoperation, and the user selects “Select as Background” from the menudisplayed after right click. The user then selects the entire input area43, and pastes the selected image therein.

The search target can be used as either a main subject or a backgroundif the target is a tree or a mountain, for example. In such a case, thetarget can be selected as either a main subject or a background.However, the image search apparatus 1 is set to paste a search targetselected as a background at the rear of a main subject. Therefore, it ispreferable for the user to appropriately use either a main subject or abackground. The user can cancel the search target by clicking the“Clear” button 45 if the target is unnecessary, even after the targethas been pasted in the input area 43.

Not only a region but also a plurality of regions may be used as thesearch image part or parts. In addition, the search conditions can beset as a combination of a plurality of main subjects, a combination ofbackgrounds, and a combination of a main subject and a background, forexample. FIG. 4 shows an example of the search image wherein the personon the left of the representative image 41A has been selected as a mainsubject A positioned on the right and a background of the representativeimage 41B has been selected as a background B in the input area 43.Since the search image can be generated by using a part or all of therepresentative image or images as the search image part or partspositioned at the desired position or positions in the desired size orsizes, fragmentary memories of the user can be faithfully and easilyvisualized.

When the user clicks the “Search” button 44 after the search image hasbeen generated, the search unit 36 searches the image group read fromthe memory card 2 for the image or images agreeing with the searchcondition (Step ST7). In the case where the search image includes a mainsubject, the search unit 36 searches the image group for the image orimages including the main subject. For example, in the case where a mainsubject is a person, the search unit 36 uses a known method such asmatching of a characteristic quantity regarding face outline and facialparts including eyes, nose, mouth, and ears. At this time, a position ofthe main subject having been found is judged, and the image or imagesshowing better agreement with the position of the main subject in thesearch image is/are searched for.

In the case where the search image includes a background, the searchunit 36 compares a color of the background with a color of a backgroundof each of the images read from the memory card 2, and finds the imageor images showing better agreement. In the case where a backgroundregion of the search image is divided into regions, that is, in the casewhere the background region comprises a plurality of pasted backgrounds,the color comparison is carried out for each of the divided regions.

In the case where the search image includes both a main subject and abackground, the search unit 36 carries out the search for the mainsubject followed by the search for the background in the same manner ashas been described above.

After the search has been completed, the display control unit 28displays a result of the search on the display unit 6 (Step ST8). FIG. 5shows an example of a screen showing the search result. In a screen 50of the search result shown in FIG. 5 is displayed a catalog of images 51extracted as the search result and displayed in order of betteragreement with the search image, a “Print” button 52 for changing to ascreen for printing, a “Retry” button 53 for searching again, and a“Quit” button 54 for ending the search to change to an initial screen.In FIG. 5, a plurality of images are collectively referred to as theimages 51.

The user selects and clicks a desired one of the buttons. If the userselects the “Print” button 52, a screen is displayed on the display unit6 for letting the user select a printing target. When the user selectsthe image to print, the CPU 12 outputs a printing instruction and sendsthe image data of the printing target to the controller 10. Thecontroller 10 carries out image processing thereon for image qualityimprovement, and the processed image data are sent to the printer 8. Theprinter 8 then outputs a photographic print.

If the user selects the “Retry” button 53, the search screen 40 isdisplayed on the display unit 6, and the user can search again. When theuser selects the “Quit” button 54, the menu screen is shown on thedisplay unit 6.

As has been described above, in this embodiment, the image search iscarried out by using a part or all of the representative image orimages. Therefore, the search can be carried out more easily and moreaccurately based on fragmentary memories remembered by the user.Furthermore, since the search conditions can include a position of amain subject or a background, the search can be carried out inconsideration of composition according to the more detailed conditions.

In the above description and in FIG. 1, the mouse is used as the inputunit 7. However, another pointing device such as a pen tablet, or akeyboard, or a touch panel can be used as the input unit 7, depending ona manner of input.

A second embodiment of the present invention is described next. FIG. 6is a block diagram showing the configuration of an image searchapparatus 1′ in the second embodiment. In the second embodiment, thesame components as in the first embodiment have the same referencecodes, and detailed description thereof is omitted.

The image search apparatus 1′ in the second embodiment comprises aregion extraction unit 38 (region extraction means) added to the imagesearch apparatus 1 in the first embodiment. Hereinafter, functions ofthe region extraction unit 38 are described together with procedurescarried out in the second embodiment.

FIG. 7 is a flow chart showing the procedures of image search in thesecond embodiment. Processing flow from Step ST11 to ST14 is the same asthe processing flow from Step ST1 to ST4 in the first embodiment.Therefore, description thereof is omitted.

After a representative image or representative images has/have beenextracted at Step ST14, the region extraction unit 38 extracts a mainsubject region and/or a background region from the representative imageor images as a candidate or candidates for selection of a part or all ofthe representative image or images by a user (Step ST15), and stores theregion or regions while causing the region or regions to becomeselectable. The region or regions can be extracted according to a knownmethod such as edge detection. Causing the extracted region or regionsto become selectable refers to displaying the extracted region orregions on the display unit 6 or displaying the extracted region orregions on the display unit 6 after adding a bounding outline, marking,and the type of region thereto, for example. The type of region refersto “person”, “background”, “object” (a region other than a person orscenery), or the like. FIG. 8 shows an example of a catalog of theextracted regions displayed on the display unit 6. FIG. 9 shows anexample displayed in a representative image 61 c wherein a person 66 hasbeen extracted as the main subject region and added with a boundingoutline 66 a and a small window 66 b representing the type of theregion. By displaying the region or regions in this manner, confirmationthereof becomes easier.

The display control unit 28 displays a search screen on the display unit6 (Step ST16). FIG. 10 shows an example of the search screen. As shownin FIG. 10, a search screen 60 includes a representative image area 62wherein a catalog of representative images 61 is shown, and an inputarea 63 for displaying a selected region. The input area 63 has a“Search” button 64 for carrying out a search and a “Clear” button 65 forclearing input data. In FIG. 10, the representative images arecollectively shown by the reference code 61, and two of the images arereferred to as images 61A and 61B.

The user selects the representative image including a search target fromthe catalog of the representative images 61, and selects as a region ofthe search target the main subject region or the background regionextracted in advance. In response, the search condition setting unit 34sets the selected region as the search conditions (Step ST17). In thisembodiment, a part or all of the selected representative image alsofunctions as a search image part. The region is selected in thefollowing manner, for example.

When the user selects the representative image in the search screen 60,a sub-menu is displayed for letting the user select either a mainsubject (region) or a background (region) in the selected representativeimage to be used as the search conditions. In the case where thebackground has been selected, the background is displayed in the inputarea 63. In the case where the main subject has been selected and therepresentative image has a plurality of main subjects, a screen isdisplayed for letting the user further select one of the main subjectsto be used as the search target. The selected main subject is thendisplayed in the input area 63. By repeating this selection procedure,the user can select a plurality of regions, and all the selected regionsare displayed in the input area 63. In the input area 63 in FIG. 10, theperson on the left of the representative image 61A has been selected asa main subject A and a background in the representative image 61B hasbeen selected as a background B. Clicking the “Clear” button 65 cancancel the selected region in the input area 63 if the region isunnecessary.

After the user clicks the “Search” button 64, the search unit 36 carriesout the search for an image or images agreeing with the searchconditions (Step ST18). The search is carried out in the same manner asin the first embodiment. However, in the second embodiment, onlymatching of the main subject and the background is carried out.Therefore, a position thereof is not considered, which is different fromthe first embodiment.

After the search is complete, the display control unit 28 displays asearch result on the display unit 6 (Step ST19). The processing flowthereafter is the same as the processing flow from Step ST8 in the firstembodiment.

As has been described above, in the second embodiment, the region orregions in the representative image or images is/are selected in advancefor letting the user carry out selection from the region or regions.Therefore, the user does not need to carry out the drag operation forregion selection and the drag-and-drop operation for pasting as in thefirst embodiment. Consequently, the user can carry out the search afterthe simple operation. For this reason, the input unit 7 in the secondembodiment does not necessarily comprise a mouse or a pen tablet, and atouch panel can be sufficient for the configuration.

The region extraction unit 38 may be applied to the first embodiment. Inthis case, a region or regions in a representative image in the firstembodiment is/are extracted in advance so that a user can generate asearch image for carrying out an image search by selection of theextracted region or regions.

A modification to the embodiments of the present invention is describednext. The configuration of the modification is the same as the secondembodiment. However, whenever a region as a search image part is set asthe search conditions, an image search is carried out to display aresult of the search, which is a difference from the second embodiment.In the description below, only the difference from the second embodimentis described, and description of the same components and the proceduresas in the second embodiment is omitted.

FIG. 11 shows an example of a search screen in the modification. Asshown in FIG. 11, a search screen 70 has a representative image area 72for display of a catalog of representative images 71, an input area 73for display of a selected region, and a search result area 74 fordisplay of a search result. In the input area 73, a “Clear” button 76for clearing input data is displayed. In the search result area 74 aredisplayed a quantity display area 77 for displaying the quantity ofimages found as the search result, and an “Image Display” button 78 fordisplaying the images found through the search.

When a user selects a main subject region or a background region as asearch image part, the region is displayed in the input area 73. Thesearch condition setting unit 34 automatically sets the selected regionas the search conditions, and the search unit 36 automatically carriesout the search. After the image search, the display control unit 28automatically displays the quantity of the images extracted through thesearch in the quantity display area 77 in the search result area 74.When the user clicks the “Image Display” button 78 at this stage, acatalog of the extracted images is displayed on the display unit 6.

Procedures from selection of the region to the display of the quantityas the search result are carried out whenever a region is selected. Forexample, when the user selects the person on the left of therepresentative image 71A as a main subject A, images including the mainsubject A are searched for, and the quantity of the images is displayedin the quantity display area 77. When the user further selects abackground of the representative image 71B as a background B, the imagesincluding the main subject A found through the previous search aresearched for images including the background B. The quantity of theimages is then displayed in the quantity display area 77.

As has been described above, according to this modification, the usercan select a region as the search image part while viewing the searchresult displayed after each search. Therefore, efficient searchrefinement can be carried out.

In the example shown in FIG. 11, only the quantity is displayed as theresult of the search. However, thumbnail images of the images found asthe search result may be displayed with or without the quantity. In theabove description, the modification has been described as themodification to the second embodiment. However, the search and thedisplay of the search result may also be carried out in the firstembodiment whenever a region as the search image part is selected. Inaddition, in the screen of the search result, the region or regionshaving been found through the search may be displayed in a state where abounding outline or bounding outlines is/are added thereto, for example.In this manner, the region or regions having been found can be easilyconfirmed.

In the embodiments and in the modification described above, the searchcondition setting unit 34 may have other functions for setting thesearch conditions. Hereinafter, two examples in this case are described.

A first example is the case where the search condition setting unit 34has a function for carrying out image processing on the part or all ofthe representative image or images having been selected as the searchimage part or parts. The search conditions are then set by using thepart or all of the representative image or images having been subjectedto the image processing. Various kinds of processing such as resizing,rotation, deformation, color change, blurring, color correction, red-eyecorrection, skin enhancement, and camera-shake correction can be listedas the image processing. For example, in the case where a user has clearmemory of one thing but has unclear memory of another thing, the thingof which the memory is unclear is blurred. In this manner, the searchconditions can be set to become more similar to the fragmentary memoriesof the user. In addition, clothes of a predetermined person, an object,or the like may be changed to have a color or texture that is moresimilar to the memory. In this case, the search conditions can also beset to become more similar to the fragmentary memories of the user.

The image processing may be carried out according to an instructioninput by the user after selection of the search image part or parts.Alternatively, the image processing may be carried out before theselection. In addition, the user may decide to execute or not to executethe image processing.

By letting the search condition setting unit 34 have the imageprocessing function, a state of the search image part or parts regardingthe content of the image or images to be found can be more appropriatelyreflected in the search.

A second example is the case where the search condition setting unit 34has a function for adding search priority as one of the searchconditions to the part or all of the representative image or imageshaving been selected as the search image part or parts. For example, thepriority is added according to what is prioritized, such as personsamong the search image parts, a specific person among a plurality ofpersons, an object (that is, a search image part other than people andscenery), a specific object among a plurality of objects, and scenery.Since the search priority is generally different for each of the searchimage parts, adding the priority can cause the search to reflect anintention of the search and a degree of clearness of memory. As aspecific method therefor, each of the search image parts is weightedaccording to the priority, and set as one of the search conditions. Thesearch is then carried out to reflect the priority in a result of thesearch.

A plurality of methods can be used as a method of adding the priority.For example, the user may input the priority for each of the searchimage parts in the form of numbers. FIG. 12 shows an example of thiscase. In the example shown in FIG. 12, the person on the right, thescenery, the house, and the person on the left pasted in an input area83 have been respectively set as search image parts A, B, C, and D. Whenthe user clicks any one of the search image parts in the input area 83,a small window 84 for inputting the priority is displayed. The user theninputs a value representing the priority by use of input means such as akeyboard.

The priority may be added to become 100 when totaled up for all thesearch image parts. In this case, a priority setting window 86 shown byFIG. 13 as an example is displayed on the screen to be used for settingthe priority. In the priority setting window 86 shown in FIG. 13, thealphabets A, B, C, and D respectively refer to the search image parts A,B, C, and D. The values “50”, “30”, “10”, and “10” above the alphabetsrespectively represent the priority set to the search image parts A, B,C, and D. The user can change the values by sliding bars 87 along thehorizontal direction of the screen. The values are set so as to total100 in any cases. The priority may be set as a ratio such as 3:3:2:1. Byadding the priority in the form of numbers, detailed setting can berealized.

Alternatively, the user may only specify the order of priority. In thiscase, the priority is set according to a ratio that is predetermined inthe order of priority. It is preferable for the ratio to be changeable.In this manner, the priority can be set easily, since only the order ofpriority needs to be specified in the case where adding detailed numberssuch as the ratio to the respective search image parts is difficult andtroublesome. Consequently, a burden on the user can be reduced.

Alternatively, modes such as a mode for prioritizing a person and a modefor prioritizing scenery may be prepared so that the user can select anyone of the modes that causes the priority to become higher for thesearch image part he/she wishes to prioritize. By using such modes, thepriority can be set more easily.

In addition, the priority may be set according to a state of each of thesearch image parts. For example, the priority may be set higher for alarger part, a less blurry part, a more foreground part when positionedin the input area, and a part occupying a larger area in thecorresponding representative image, for example. By using such setting,easiness of operation is improved and the priority can be set moreeasily according to sensation of the user.

The function for carrying out the image processing described as thefirst example may be used together with the function for adding thepriority described as the second example. For example, when apredetermined search image part is subjected to blurring carried out bythe image processing function and added with the priority, the prioritymay be set to become higher as a degree of blur becomes lower.Furthermore, in the case where a size of a predetermined search imagepart is enlarged by the image processing function and the priority isadded thereto, the priority may be set higher as the size thereofbecomes larger.

Various modifications can be made to the present invention, in additionto the embodiments and the modifications described above. For example,the search result is displayed for all the search image parts in theorder of better agreement in the embodiments described above. However,the search result may be displayed in order of better agreement for eachof the search image parts. FIG. 14 shows an example in this case. In ascreen 90 representing a search result in the example shown in FIG. 14,the search image parts A, B, C, and D are shown vertically in theleftmost column. Images 91 arranged in the order of better agreement areshown as the search result, in rows to the right of the respectivesearch image parts.

In the above embodiments, the case has been described where an imageobtained as the search result is printed. However, use of the searchresult is not necessarily limited to printing. The image data obtainedthrough the search result may be sent to a desired location or writtenin a desired recording medium.

When the representative image or images is/are extracted, the respectiveimages may be classified according to date based on information onphotography time and date thereof so that an image that best representsthe type of event or main subject can be extracted as the representativeimage according to the classification. In addition, an access log suchas past searches and printing may be stored so that an image that hasbeen frequently ranked high in the past searches or frequently printedcan be extracted as the representative image.

The representative image from which the search image part has beenextracted has a substantially high probability of being found in asearch using the part. Therefore, the priority added thereto may belowered or the image may be excluded at the time of the search.

Although the embodiments of the present invention and the modificationsthereto have been described above, a program that causes a computer tofunction as means corresponding to the representative image extractionunit 32, the search condition setting unit 34, the search unit 36, andthe region extraction unit 38 for carry out the procedures shown in FIG.3 or 7 is also an embodiment of the present invention. Acomputer-readable recording medium storing the program is also anembodiment of the present invention.

What is claimed is:
 1. A computer implemented method comprising:selecting, by a computing system, a first image portion of a firstimage, wherein the first image portion reflects at least one type of amain subject or a background; in response to instructions provided by auser, performing, by the computing system, image processing on a firstcomponent of the first image portion to generate a processed first imageportion; performing, by the computing system, a first search on imagedata of the processed first image portion, wherein the first searchgenerates a first set of search results based on the selected type;selecting, by the computing system, a second image portion of a secondimage, wherein the second image portion reflects at least one type of amain subject or a background different from the selected type for thefirst image portion; in response to instructions provided by the user,performing, by the computing system, image processing on a secondcomponent of the second image portion to generate a processed secondimage portion; and performing, by the computing system, a second searchon the image data of the processed second image portion, wherein resultsof the second search generates a second set of search results that is asubset of the first set of search results.
 2. The computer implementedmethod of claim 1, wherein the image processing includes at least one ofresizing, rotation, deformation, color change, blurring, colorcorrection, red-eye correction, skin enhancement, or camera-shakecorrection.
 3. The computer implemented method of claim 1, wherein theimage processing is based at least in part on memory of the userregarding the first component or the second component.
 4. The computerimplemented method of claim 1, wherein the image processing is carriedout after the selecting the first image portion of the first image. 5.The computer implemented method of claim 1, wherein the image processingis carried out before the selecting the first image portion of the firstimage.
 6. The computer implemented method of claim 1, wherein the firstimage portion is assigned a first weight, the second image portion isassigned a second weight, and the performing a search on the image datais based at least in part on the first weight and the second weight. 7.The computer implemented method of claim 6, wherein the first weight orthe second weight is based at least in part on at least one of a size, aposition, and a degree of blurriness of the first image portion or thesecond image portion, respectively.
 8. The computer implemented methodof claim 6, wherein the first weight and the second weight reflect arelative priority of, respectively, the first image portion and thesecond image portion for the search.
 9. The computer implemented methodof claim 6, wherein the first weight and the second weight are based ona predetermined ratio.
 10. The computer implemented method of claim 1,wherein the selecting the first image portion of the first image and thesecond image portion of the second image is performable by the user. 11.A system comprising: at least one processor; and a memory storinginstructions that, when executed by the at least one processor, causethe system to perform: selecting a first image portion of a first image,wherein the first image portion reflects at least one type of a mainsubject or a background; in response to instructions provided by a user,performing image processing on a first component of the first imageportion to generate a processed first image portion; performing a firstsearch on image data of the processed first image portion, wherein thefirst search generates a first set of search results based on theselected type; selecting a second image portion of a second image,wherein the second image portion reflects at least one type of a mainsubject or a background different from the selected type for the firstimage portion; in response to instructions provided by the user,performing image processing on a second component of the second imageportion to generate a processed second image portion; and performing asecond search on the image data of the processed second image portion,wherein results of the second search generates a second set of searchresults that is a subset of the first set of search results.
 12. Thesystem of claim 11, wherein the image processing includes at least oneof resizing, rotation, deformation, color change, blurring, colorcorrection, red-eye correction, skin enhancement, or camera-shakecorrection.
 13. The system of claim 11, wherein the image processing isbased at least in part on memory of the user regarding the firstcomponent or the second component.
 14. The system of claim 11, whereinthe image processing is carried out after the selecting the first imageportion of the first image.
 15. The system of claim 11, wherein thefirst image portion is assigned a first weight, the second image portionis assigned a second weight, and the performing a search on the imagedata is based at least in part on the first weight and the secondweight.
 16. A non-transitory computer-readable storage medium includinginstructions that, when executed by at least one processor of acomputing system, cause the computing system to perform a methodcomprising: selecting a first image portion of a first image, whereinthe first image portion reflects at least one type of a main subject ora background; in response to instructions provided by a user, performingimage processing on a first component of the first image portion togenerate a processed first image portion; performing a first search onimage data of the processed first image portion, wherein the firstsearch generates a first set of search results based on the selectedtype; selecting a second image portion of a second image, wherein thesecond image portion reflects at least one type of a main subject or abackground different from the selected type for the first image portion;in response to instructions provided by the user, performing imageprocessing on a second component of the second image portion to generatea processed second image portion; and performing a second search on theimage data of the processed second image portion, wherein results of thesecond search generates a second set of search results that is a subsetof the first set of search results.
 17. The non-transitorycomputer-readable storage medium of claim 16, wherein the imageprocessing includes at least one of resizing, rotation, deformation,color change, blurring, color correction, red-eye correction, skinenhancement, or camera-shake correction.
 18. The non-transitorycomputer-readable storage medium of claim 16, wherein the imageprocessing is based at least in part on memory of the user regarding thefirst component or the second component.
 19. The non-transitorycomputer-readable storage medium of claim 16, wherein the imageprocessing is carried out after the selecting the first image portion ofthe first image.
 20. The non-transitory computer-readable storage mediumof claim 16, wherein the first image portion is assigned a first weight,the second image portion is assigned a second weight, and the performinga search on the image data is based at least in part on the first weightand the second weight.